Skip to content

DEV-1242: Android HIL tests on Shimmer3/3R over Classic and BLE, and a release gate - #128

Merged
JongChern merged 22 commits into
masterfrom
DEV-1242_android_hil
Oct 23, 2026
Merged

JongChern merged 22 commits into
masterfrom
DEV-1242_android_hil

Conversation

@JongChern

Copy link
Copy Markdown
Contributor

What this adds

Hardware-in-the-loop tests of the Android API. A phone on USB connects to real Shimmer3 and Shimmer3R units and drives them through the same public API an app uses (ShimmerBluetoothManagerAndroid, ShimmerBluetooth). One command builds the tests, installs them, runs every scenario on every unit and transport, pulls the reports and prints pass/fail:

.\tools\hil\run-android-hil.ps1

Setup, the scenarios, and how to read the reports are in tools/hil/README.md.

  • shimmerHilTest: a new module of instrumented tests, driven via am instrument.
    • Scenarios:
      • streamRest: stream with the unit at rest, then check loss, timestamps, effective rate and accel.
      • midStreamSet: start and stop SD logging and read the battery, all while streaming.
      • reconnectLoop: connect, stream and disconnect, five times.
    • Judged from the device's own timestamps (StreamAnalysis, with JVM unit tests). It tells lost packets apart from a bad timestamp that leaps ahead of its arrival time.
    • Watchdog: any API call that doesn't return within 10 s fails the scenario, and the report gets every thread's stack. A phone without root gives no other way to see a deadlock.
  • Both transports, per unit, because on Android they run different code:
    • Classic goes through the Android API's own Shimmer class and its copy of the parser;
    • BLE goes through Shimmer3BLEAndroid and the driver's parser.
  • tools/hil/run-android-hil.ps1: the runner.
    • Options: -Unit, -Transport, -Scenario, -Crc, -DriverVersion, -AndroidApiVersion (test the published .aar), -SettleSeconds.
    • Saves the logcat and instrument output for every run.
    • Force-stops a run that hangs.
  • Release gate: -Report posts the commit status android-hil (driver X) on the commit it tested.
    • It refuses partial runs: every scenario, at least one Shimmer3 and one Shimmer3R, both transports, a clean tree, and a commit that's on GitHub.
    • Release Android API reads that status for the commit and driver it's releasing. It stops unless the status passed, or hil_waiver holds a Jira key. Dry runs only warn.
    • The release notes record what was tested, or the waiver.

Heads-up: releases will need a waiver for now

The HIL already finds real defects, so a full -Report run can't pass yet. Until these are fixed, a release needs hil_waiver set to a Jira key:

Finding Ticket
Classic, 2-byte CRC: ~6% of packets silently lost (0% with CRC off, 0% over BLE) DEV-1248
Classic, CRC on: SD logging never starts while streaming DEV-1148 (#121)
Driver lock inversion (IO thread vs TimerCheckForResp), caught on hardware with a thread dump DEV-1198
Shimmer3R: packets after an SD-logging start are stamped out of order; one turned into a 512 s timestamp jump in the driver DEV-1245
Shimmer3 v1.1.5 stops answering every command after a BLE config write DEV-1246
Every BLE connect blocks the main thread ~3 s DEV-1247

Tested

  • StreamAnalysisTest: 8 of 8 pass on the JVM.
  • On a Xiaomi phone (Android 11) with Shimmer3 E8:EB:1B:71:3E:36 (v1.1.5, Classic and BLE) and Shimmer3R EC:06:D8:0D:86:F8 (v1.1.21, BLE), driver 0.11.9_beta.
    • Shimmer3R BLE: streamRest and reconnectLoop pass. midStreamSet fails on DEV-1245.
    • Shimmer3: Classic fails on DEV-1248 and DEV-1148. BLE passed streamRest (apart from this unit's accel calibration, which reads 7.58 m/s²) until the unit hung (DEV-1246).
  • The release workflow's HIL check was run against real GitHub statuses, and against a mock gh for: pass, newest status failed, a different driver, and pending.
  • Not covered: the Shimmer3R over Classic. This phone makes an LE bond with it, so the RFCOMM channel is refused (DEV-970).

🤖 Generated with Claude Code

JongChern and others added 3 commits October 20, 2026 11:05
- shimmerHilTest: instrumented tests that drive the Android API on a
  USB-attached phone against real units over Bluetooth Classic.
  stream_rest (loss, timestamps, rate, accel at rest, parser errors),
  mid_stream_set (SD logging and battery reads while streaming: the
  DEV-1148 regression) and reconnect_loop. Each run writes a JSON
  report and, for stream_rest, a CSV of samples. The stream analysis
  is plain Java with JVM unit tests.
- tools/hil/run-android-hil.ps1: one command builds against any
  -PdriverVersion (and optionally the published .aar), installs, runs
  every scenario on every unit in a local rig file, pulls the reports
  and prints pass/fail.
- Release gate: -Report posts the commit status
  "android-hil (driver X)" for a full run of a clean, pushed commit.
  Release Android API refuses to build a commit without a passing one
  for its driver, unless hil_waiver has a Jira key, and puts the
  result in the release notes.
- tools/hil/README.md documents setup and the release step.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Found running the tests on a Redmi Note 10 Pro (Android 11) against a
Shimmer3 (LogAndStream v1.1.5):
- Build: tell Gradle where the SDK is when there's no local.properties.
- Install: MIUI refuses adb install -g; up to Android 11 the tests
  need no runtime permission, so install without it there.
- ShimmerBluetoothManagerAndroid creates a Handler as it is constructed,
  so construct it on the main thread, as an app does.
- Configure: the progress report doesn't reach the app's Handler; treat
  the fresh "Ready" with the rate read back as done (20 s -> 3 s).
- midStreamSet: charge each gap to the nearest command, and report but
  don't fail loss around SD logging start/stop (the unit pauses while it
  opens its file; firmware behaviour).
- A watchdog on every API call: a call that never returns fails the
  scenario with every thread's stack in the report. The runner also
  force-stops a scenario that runs past -ScenarioTimeoutSeconds.
- Reports carry the full gap list; the README covers the above and the
  DEV-970 Shimmer3R bonding symptom.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
…st link

- Each rig unit lists the transports to test it over; the runner runs every
  unit over each, and -Report requires both, since on Android they are
  different parsers (Classic: the Android API's Shimmer class; BLE: the
  driver's, through Shimmer3BLEAndroid).
- StreamAnalysis tells a timestamp that leaps far ahead of its arrival time
  (a bad unwrap, DEV-1023 class) from lost packets, and reports both.
- midStreamSet writes its samples to CSV, and the report lists every gap and
  jump with when it arrived.
- The session no longer throws when the API drops a unit whose link was lost;
  calls fail as "connection lost" instead.
- -SettleSeconds (default 10) sets the pause between runs.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Copilot AI lite review requested due to automatic review settings October 20, 2026 04:43

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Moderate issues can allow failed tests or invalid results to pass and unblock releases.

5 open issues
What changed in this PR

Adds Android hardware-in-the-loop testing for Shimmer3/Shimmer3R over Classic and BLE, with reporting and a release-status gate.

Changes:

  • Adds instrumented HIL scenarios and stream-analysis tests.
  • Adds runner tooling, reports, configuration, and documentation.
  • Integrates HIL status and waiver handling into releases.
File Summary
tools/​hil/​run-android-hil.ps1 Runs and reports HIL tests; report mode must prevent stale builds and test-shaping overrides.
tools/​hil/​rig-android.example.json Provides sample rig configuration.
tools/​hil/​README.md Documents HIL setup, scenarios, and release gating.
ShimmerAndroidInstrumentDriver/​shimmerHilTest/​src/​test/​java/​com/​shimmerresearch/​hiltest/​StreamAnalysisTest.java Tests stream-analysis calculations.
ShimmerAndroidInstrumentDriver/​shimmerHilTest/​src/​main/​java/​com/​shimmerresearch/​hiltest/​StreamAnalysis.java Analyzes packet loss, timestamps, rate, and acceleration.
ShimmerAndroidInstrumentDriver/​shimmerHilTest/​src/​main/​AndroidManifest.xml Declares HIL application metadata and permissions.
ShimmerAndroidInstrumentDriver/​shimmerHilTest/​src/​androidTest/​java/​com/​shimmerresearch/​hiltest/​ShimmerSession.java Drives Android API sessions; watchdog coverage must include condition evaluation.
ShimmerAndroidInstrumentDriver/​shimmerHilTest/​src/​androidTest/​java/​com/​shimmerresearch/​hiltest/​HilScenariosTest.java Implements scenarios; failed SD logging, battery reads, and unavailable rates must fail runs.
ShimmerAndroidInstrumentDriver/​shimmerHilTest/​src/​androidTest/​java/​com/​shimmerresearch/​hiltest/​HilReport.java Writes scenario reports and sample data.
ShimmerAndroidInstrumentDriver/​shimmerHilTest/​build.gradle Configures the HIL test module.
ShimmerAndroidInstrumentDriver/​settings.gradle Includes the HIL module.
README.md Links to Android HIL documentation.
.gitignore Ignores local HIL configuration and output.
.github/​workflows/​release.yml Adds HIL release gating and waiver support.

Review effort: Lite

Comment thread tools/hil/run-android-hil.ps1 Resolved
Comment thread tools/hil/run-android-hil.ps1 Resolved
…tings

- midStreamSet records a failure when startSDLogging, stopSDLogging or
  readBattery throws, hangs or is skipped, instead of skipping the state
  check and possibly passing.
- Every scenario fails when the API has no sampling rate for the unit,
  since all stream checks are relative to it.
- -Report refuses -Rate, -Seconds, -Crc, -MaxLossPercent and -SkipBuild:
  the gate's status must come from a fresh build at the default criteria.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Copilot AI lite review requested due to automatic review settings October 20, 2026 04:53

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Seven unresolved moderate findings affect report reliability, scenario validation, timing, loss accounting, hardware coverage, and release-status integrity.

No open issues 🎉

1 previously missed issue

In code that hasn't changed since last review

Medium severity NaN effective rate incorrectly skips rate validation

ShimmerAndroidInstrumentDriver/​shimmerHilTest/​src/​androidTest/​java/​com/​shimmerresearch/​hiltest/​HilScenariosTest.java:157

When only one packet is received, or the timestamp channel is unavailable/NaN, StreamAnalysis leaves effectiveRateHz as NaN and this conditional skips the rate assertion. A run with a single valid accel packet can therefore pass despite having no usable device-timestamp stream. Treat a non-finite effective rate as a failure instead of skipping this check.

5 resolved since last review

Review effort: Lite

…lent

Loss is counted between packets received, so two cases passed unseen:
- fewer than two usable device timestamps left the effective rate NaN, and
  the rate check was skipped; it now fails;
- a stream that stopped early left no gap between the packets it did
  deliver; now every scenario fails if no packet arrived in the last second
  before recording stopped.
midStreamSet gets the same rate and silence checks as the other two.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Copilot AI lite review requested due to automatic review settings October 20, 2026 05:03

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Moderate reliability issues can allow incomplete or invalid HIL results to pass the release gate.

No open issues 🎉

1 previously missed issue

In code that hasn't changed since last review

Medium severity Ignored adb pull failure allows false PASS status

tools/​hil/​run-android-hil.ps1:297

The report pull's exit status is ignored, so an adb pull failure can still leave all instrumentation results as PASS and post a successful release-gate status without any reports collected. Capture the command result and fail the run before posting the status when the reports cannot be retrieved.

Review effort: Lite

The report pull's result was ignored, so a failed adb pull could leave
every run PASS and -Report post a success with no reports behind it. Each
run that reached a verdict must now have its report on the PC; if not, it
becomes ERROR (with the adb output when the pull failed), and the posted
status is a failure.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Copilot AI lite review requested due to automatic review settings October 20, 2026 05:10

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Four moderate issues remain unresolved in test validation and result reporting.

No open issues 🎉

1 previously missed issue

In code that hasn't changed since last review

Medium severity Count skipped tests as non-passing in final exit status

tools/​hil/​run-android-hil.ps1:324

A skipped instrumentation test is not counted in $failed, so a run containing SKIP reaches the final success message and exits 0 ($failed only selects values other than PASS and SKIP). This contradicts the documented contract that exit 0 means everything passed and can make unattended/non--Report runs look successful despite scenarios not running; treat SKIP as a non-pass in the final result (the report gate already does this when posting status).

Review effort: Lite

The runner exited 0 and printed "All N passed" when a scenario was
skipped, against the README's "0 when everything passed". SKIP now counts
as not passed for the exit code and the closing message, as it already did
for the release-gate status.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Copilot AI lite review requested due to automatic review settings October 20, 2026 05:19

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Moderate unresolved correctness issues remain in reporting, battery verification, packet capture synchronization, and runner validation.

No open issues 🎉

3 previously missed issues

In code that hasn't changed since last review

Medium severity Non-finite metrics abort stream report generation

ShimmerAndroidInstrumentDriver/​shimmerHilTest/​src/​androidTest/​java/​com/​shimmerresearch/​hiltest/​HilReport.java:69

JSONObject rejects NaN and infinite numbers. A run with no usable timestamps or no calibrated accel (both are explicitly handled as failures by the scenario) therefore throws while building this object and silently omits the entire stream section from the report, losing the diagnostic values for the failing run. Store non-finite metrics as JSONObject.NULL (or serialize them explicitly) instead of allowing putStream to abort.

Medium severity Battery command success is not actually verified

ShimmerAndroidInstrumentDriver/​shimmerHilTest/​src/​androidTest/​java/​com/​shimmerresearch/​hiltest/​HilScenariosTest.java:284

readBattery() is a void, fire-and-forget API call (the Classic implementation only calls writeBytes and returns), so this assertion only proves that the write invocation returned. No callback/value is recorded or awaited, meaning a dropped/unanswered battery command can still produce a passing midStreamSet run as long as streaming continues. Capture the battery response (or a corresponding callback/event) and wait for it with the same timeout before treating this cycle as successful.

Medium severity Ignored cleanup failure allows stale reports

tools/​hil/​run-android-hil.ps1:229

The cleanup result is ignored. If rm -rf fails, reports from an earlier run remain on the phone, and the later prefix-only lookup can accept one of those stale files as evidence for a current PASS. Check $LASTEXITCODE after this command and abort before running when the report directory cannot be cleared.

Review effort: Lite

…anup

- midStreamSet now waits up to 5 s for each battery reply to be decoded
  (the API swaps in a new battery-details object per reply), not just for
  readBattery(), which only sends the command, to return.
- HilReport stores NaN and infinite metrics as null. org.json refuses
  them, which dropped the whole stream section from exactly the failing
  runs that need it.
- The runner checks that the phone's old reports were cleared before the
  run, so a stale report cannot stand in for a missing one.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Copilot AI lite review requested due to automatic review settings October 20, 2026 05:30

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Three unresolved moderate findings affect report accuracy and release-gate correctness.

No open issues 🎉

Review effort: Lite

…ytes)

A HIL finding is only fixed for good once its bytes are a byte-level test.
With -CaptureBytes the runner notes the phone's clock around each run, takes
one bug report at the end (the only way adb can read the HCI snoop log on an
unrooted phone), keeps btsnoop_hci.log, deletes the bug report from the PC and
the phone, and cuts out each run's bytes by the unit's MAC and time window.

btsnoop_to_trace.py writes shimmer-logandstream's trace format
(tests/data/*.bytes.log): BLE ATT writes and notifications, and Classic RFCOMM
payloads as the radio delivered them, with L2CAP fragments reassembled. It is
standard-library Python with unit tests on a synthetic snoop log. Capturing
is diagnostic only and never changes a verdict.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
… force

Found on the first real capture (Xiaomi, Android 11):
- Android stamps btsnoop_hci.log in the phone's local time, so every run's
  window missed its bytes by the UTC offset. The extractor now takes the log
  as local time by default (--snoop-clock utc for one that isn't).
- adb cannot read persist.bluetooth.btsnooplogmode on Android 11, so the
  runner asks the Bluetooth service which mode it started with
  (mSnoopLogSettingAtEnable), which is also the one actually in force.
- summary.json keeps each run's capture window and the phone's UTC offset,
  so the bytes can be cut out of btsnoop/ again later.

Verified on a Shimmer3R over BLE: every sample the API delivered in
streamRest (7607) and midStreamSet (5273) is a CRC-valid packet in the
captured bytes, in the same order.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
JongChern and others added 2 commits October 20, 2026 15:25
…e bug report

- The notification-on write is matched only against notifications on the
  same connection: ATT handles belong to one connection, so after a
  reconnect a real 2-byte command could be dropped (new test; it fails on
  the old filter).
- If the bug report can't be read, say so and still delete it from the PC
  and the phone; it holds the phone's whole state.
- The time-window test now checks both ends at the record that completes
  the notification.
- summary.json's extra fields use Add-Member's named parameters (the
  positional form worked; this is for readability).

Co-Authored-By: Claude Opus 5.5 <[email protected]>
A review read the second length octet as carrying an EA bit. It doesn't:
Linux sends a long length as len << 1, little-endian over two octets
(__len16), so 130 is [0x04, 0x01] and the high octet is len >> 7 whole, as
the extractor decodes it. Comments now say so, and the test builds the
length the way Linux does. No change in behaviour.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
JongChern and others added 4 commits October 22, 2026 13:55
inStream (DEV-1291): while streaming, start SD logging, stop it, and read the
battery and the status every 40 ms for 400 ms. Android writes those reads
straight out, so some land in the ~100 ms the unit pauses its data packets
while it closes its SD file, and their replies come with no data packet
before them. Then stop and start logging back to back, so the queued start
goes out in the pause. It fails when:
- a battery read goes unanswered, or fewer status replies than reads are decoded;
- logging doesn't stop or restart;
- a command times out;
- there's a parser error.
Loss near the reads is reported, not failed: a Shimmer3 skips a sample there
now and then on its own.

connectReady (DEV-1275): connect to the unit idle, then again while it logs.
It fails when the unit is reported ready (a CONNECTED or SDLOGGING state
message, or that state in the API) more than 250 ms before
NOTIFICATION_SHIMMER_FULLY_INITIALIZED, which ends the connect sequence. The
HIL's own connect waits for isInitialised(), which on Classic comes only at
the end, so the existing scenarios never saw the early report.

The runner runs both by default, and the README lists them.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
…s lock

The IO thread takes the queue's lock and then the device's (DEV-1198). Polling
the queue through the synchronized getListofInstructions() made the IO thread
wait on the device's lock while it held the queue's, which widened that
deadlock's window. Read the field directly instead.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
…stalled connect

connectReady's second connect, to the unit while it logs, can fail. The unit
was then left logging, and every later run failed to connect to it as idle.
It now connects again to stop logging, and resetUnit does the same by hand.

A connect that doesn't finish now dumps every thread, as a hung call does.
The connect call itself returns at once, so nothing showed a stalled connect
sequence, such as the DEV-1198 lock-order deadlock.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
…at an SD start are the unit's

The phone's log buffer is small, and with the Bluetooth snoop log on it rolled
past inStream's start marker within a run. Every count then read 0, so 15
decoded battery replies were reported as none. inStream now places a marker
per round and counts right after it. If a marker is gone anyway, it fails
saying the lines could not be counted, rather than counting none.

A timestamp jump within 3 s after an SD logging start or stop is now reported
rather than failed, just as the loss there is. The captured bytes show
a Shimmer3 doing this itself. After a 1.5 s pause while it opened its file, it
sent two packets it had held, and a millisecond later the next one it
sampled, 385 periods on.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
The driver guards its instruction queue with two locks, the device and the
queue. Up to 0.11.10_beta its IO thread took them queue first when it
fetched the next instruction, and every call that queues one takes them
device first, so the two could deadlock and every later call hung. On
Android it hung 5 of 12 inStream runs on driver 0.11.7_beta, but inStream
needs SD logging and catches it only by chance.

queueStress makes the window wide on purpose, as the Java HIL's scenario of
the same name does. With the unit connected and idle, timers on, four
threads queue battery and status reads in tight loops for 60 s, keeping the
queue under 8. Idle, because while streaming Android writes these reads
straight out, unqueued, and they never take the queue's lock. It needs no
SD card, so it also runs on a docked or charging unit.

It fails when a read doesn't return within 10 s or no read returns for 10 s,
with every thread's stack in the events; when a read throws; or when the
queue doesn't drain within 15 s. The runner runs it by default and the
README lists it.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Up to driver 0.11.10_beta, a configuration written straight after
stopStreaming() could lose its first chunk, the one holding the sampling
rate: operationPrepare() cleared the instruction list while the STOP was
still in flight, and the stop's reply then retired the next instruction in
its place. The unit kept its old rate. The Java HIL's samplingRates found it
on a Shimmer3 and a Shimmer3R. Android configures through the same driver
code over both transports, and had no scenario for it.

samplingRates, as in the Java and shimmer-logandstream HILs, writes 51.2, 256
and 512 Hz in turn and streams 5 s at each, then writes the unit's own rate
back; every configure but the first comes straight after a stop. Unlike the
Java HIL's configure, the Android one can finish on the manager's SUCCESS
without looking at the rate, so the scenario checks that the rate read back
is the one written. No SD card is needed. The runner runs it by default and
the README lists it.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
…pling_rates

DEV-1306: Android HIL samplingRates, configuring straight after a stop
…ue_stress

DEV-1266: Android HIL queueStress, to reproduce the lock-order deadlock
…narios

DEV-1291 DEV-1275: Android HIL inStream and connectReady scenarios
DEV-1251: HIL saves each run's Bluetooth bytes as a replayable trace (-CaptureBytes)
@JongChern
JongChern merged commit 5f25222 into master Oct 23, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants